As the processing power of microprocessors and the quality of graphics systems have increased, advanced imaging functions such as environment mapping have become feasible on personal computer systems. Environment mapping systems use computer graphics to display the surroundings or environment of a theoretical viewer. Ideally, a user of the environment mapping system can view the environment at any angle or elevation. FIG. 1(A) illustrates the construct used in conventional environment mapping systems. A viewer 105 is centered at the origin of a three dimensional space having x, y, and z coordinates. The environment of viewer 105 (i.e., what the viewer can see) is ideally represented by a sphere 110, which surrounds viewer 105. Generally, for ease of calculation, sphere 110 is defined with a radius of 1 and is centered at the origin of the three dimensional space. More specifically, the environment of viewer 105 is captured and then re-projected onto the inner surface of sphere 110. Viewer 105 has a view window 130 which defines the portion of sphere 110 viewer 105 can see at any given moment. View window 130 is typically displayed on a display unit for the user of the environment mapping system.
In many situations, the portions of an environment above and below user 105 are not relevant. For example, viewer 105 is standing in a building, details of the floor and ceiling may not be necessary. Thus, many environment mapping systems ignore these areas and use a cylindrical representation of the environment around user 105. FIG. 1(B) is a simplified graphical representation of a cylindrical texture projection of an environment surrounding viewer 105 in a second conventional environment mapping system. Viewer 105 is centered at the origin of a three dimensional space having x, y, and z coordinates. The environment of viewer 105 (i.e., what the viewer can see) is ideally represented by a cylinder 150, which surrounds viewer 105. Generally, for ease of calculation, cylinder 150 is defined with a radius of 1 and is centered at the origin of the three dimensional space. The height of cylinder 150 varies with different environment mapping systems. More specifically, the environment of viewer 105 is captured and then re-projected onto the inner surface of cylinder 150. Viewer 105 has a view window 130 which defines the portion of cylinder 150 viewer 105 can see at any given moment. View window 130 is typically displayed on a display unit for the user of the environment mapping system.
Conventional environment mapping systems include an environment capture system and an environment display system. The environment capture system creates an environment map which contains the necessary data to recreate the environment of viewer 105. The environment display system displays portions of the environment in view window 130 based on the field of view of the user of the environment display system. An environment display system is described in detail by Hashimoto et al., in co-pending U.S. patent application Ser. No. 09/505,337, entitled “POLYGONAL CURVATURE MAPPING TO INCREASE TEXTURE EFFICIENCY.” Typically, the environment capture system includes a camera system to capture the entire environment of viewer 105. Specifically, the field of view of the camera system must encompass the totality of the inner surface of sphere 110 or cylinder 150.
An extension to environment mapping is generating and displaying immersive videos. Immersive videos involve creating multiple environment maps, ideally at a rate of at least 30 frames a second, and displaying appropriate sections of the multiple environment maps for viewer 105, also ideally at a rate of at least 30 frames a second. Immersive videos are used to provide a dynamic environment rather than a single static environment as provided by a single environment map. Alternatively, immersive video techniques allow the location of viewer 105 to be moved. For example, an immersive video can be made to capture a flight in the Grand Canyon. The user of an immersive video display system would be able to take the flight and look out at the Grand Canyon at any angle. Camera systems for environment mappings can be easily converted for use with immersive videos by using video cameras in place of still image cameras.
Many conventional camera systems exist to capture the entire environment of viewer 105. For example, cameras can be adapted to use hemispherical lens to capture a hemisphere of sphere 110, i.e. half of the environment of viewer 105. By using two camera with hemispherical lens the entire environment of viewer 105 can be captured. However, the images captured by a camera with a hemispherical lens require intensive processing to remove the distortions caused by the hemispherical lens. Furthermore, two-cameras systems provide less resolution than systems having more than two cameras.
Other environment capturing systems use multi-camera capture systems. FIG. 2(a) depicts an outward facing camera system 200 having six cameras 211–216 facing outward from a center point C. Outward facing camera system 200 captures a cylindrical portion of the environment. Camera 211 is directed to capture data representing a region 221 of the environment surrounding camera system 200. Similarly, cameras 212–216 are directed to capture data representing regions 222–226, respectively. The data captured by cameras 211–216 is then combined in a cylindrical environment display system (not shown) to create a corresponding environment map from the perspective of viewer 105.
As shown in FIG. 2(b), outward facing camera system can be used to capture spherical environments by adding cameras to capture the portions of the environment above and below the portions captured by the cameras shown in FIG. 2(a). For example, FIG. 2(b) shows an outward facing camera system 250 having ten cameras 261–270 facing outward from a center point (not shown). Camera 270 is obstructed by camera 262 and thus not shown in FIG. 2(b). Outward facing camera system 250 captures a spherical environment. Cameras 261, 262, 264 and 270 (not shown) capture a cylindrical portion of the spherical environment. Cameras 264, 265, and 266 capture the portions of the spherical environment below the cylindrical portion captured by cameras 261, 262, 263 and 270. Similarly, cameras 267, 268, and 269 capture the portions of the spherical environment above the cylindrical portion captured by cameras 261, 262, 263 and 270. The data captured by cameras 261–270 is then combined in a spherical environment display system (not shown) to create a corresponding environment map from the perspective of viewer 105.
Another camera system for cylindrical environment capture is illustrated in FIGS. 3(a) and 3(b). Specifically, FIG. 3(a) shows a cross-sectional view of a pyramidal mirror camera system 300. Pyramidal mirror camera system 300 includes a mirror mounting bar 310, a top plate 320, a plurality of mirrors including a mirror 332 and a mirror 336, a camera mounting bar 340, and a plurality of cameras including a camera 352 and a camera 356. Top plate 320 is attached to mirror mounting bar 310. The plurality of mirrors including mirrors 332 and 336 are attached to top plate 320 and mirror mounting bar 310 at a 45 degree angle with respect to mirror mounting bar 310. The plurality of mirrors form a pyramid shape. The specific number of mirrors may differ. For example, pyramidal-mirror camera systems using 9 mirrors, 8 mirrors and 4 mirrors have been developed.
The plurality of cameras, including cameras 352 and 356 are mounted to camera mounting bar 340, which is attached to mirror mounting bar 310, so that the plurality of cameras point toward the plurality of mirrors. Specifically, each camera is aligned with one mirror so that the mirror redirects the field of view of the aligned camera outward from the pyramid formed by the mirrors. Typically, the field of view of a camera is given as an angular measurement. However, for clarity field of views are shown in the figures as projections from the nodal point of the camera. For example, the field of view of camera 352 is redirected by mirror 332 to form a field of view 362. Similarly, the field of view of camera 356 is redirected by mirror 336 to form a field of view 366. By surrounding mirror mounting bar 310 and camera mounting bar 340 with aligned mirrors and cameras, respectively, a 360 degree environment surrounding mirror mounting bar 310 can be captured by pyramidal mirror camera system 300.
FIG. 3(b) shows a view of pyramidal mirror camera system 300 from beneath the pyramid formed by the mirrors but above the plurality of cameras. Specifically, FIG. 3(b) shows mirrors 331–338 coupled to top plate 320 and the bottom of mirror mounting bar 310. Mirrors 331–338 are trapezoids rather than triangles to facilitate mounting of mirrors 331–338 to mirror mounting bar 310. Other pyramidal camera systems may use triangular mirrors. FIG. 3(b) also shows the reflection of lens 351L–358L of cameras 351–358 (not shown in FIG. 3(b)) to illustrate the alignment of the mirrors and the cameras.
Multi-camera capture systems can also be used to capture a scene from multiple angles. For example, a boxing match can be captured by a multi-camera capture system so that the boxing match can be seen from any angle. FIG. 4 illustrates an inward-facing camera system 400 for capturing a scene from multiple angles. Specifically, inward-facing camera system includes cameras 411, 412, 413, 414, 415, and 416, which all face towards a center point C. Any event occurring at or near center point C is captured by cameras 411–416 at multiple angles.
The cameras used in multi-camera capture systems typically includes a video capture system, an audio capture system, and a recording system. The video capture system and audio capture system provides video and audio data, respectively, to the recording system. The recording system is typically a video recorder that stores the video stream and audio signal onto a video medium such as a video tape. Generally, the audio signal is stored in an audio tract of the video medium. Some cameras provide multiple audio capture systems to capture multiple audio signals, e.g. a left audio signal and a right audio signal for stereo systems. In some cameras, the recording system is packaged with the video capture system and the audio capture system. In other cameras, the recording system is coupled the video capture system and audio capture system using wires.
A major problem with multi-camera capture systems is synchronization of the video streams from the multiple cameras. Some multi-camera capture systems are equipped with time code systems to provide synchronization. However, generally time code systems are only available on large expensive multi-camera capture systems, which are not suited for many applications. For example, outward facing camera systems should be made small to reduce parallax. Hence there is a need for a method and system for synchronizing the video streams from the cameras of a multi-camera capture system.